
  Builtin Tokens — Pygments
  
  
    body {
    background-color: #f2f2f2;
    margin: 0;
    padding: 0;
    font-family: 'Georgia', serif;
    color: #111;
}

#content {
    background-color: white;
    padding: 20px;
    margin: 20px auto 20px auto;
    max-width: 800px;
    border: 4px solid #ddd;
}

h1 {
    font-weight: normal;
    font-size: 40px;
    color: #09839A;
}

h2 {
    font-weight: normal;
    font-size: 30px;
    color: #C73F00;
}

h1.heading {
    margin: 0 0 30px 0;
}

h2.subheading {
    margin: -30px 0 0 45px;
}

h3 {
    margin-top: 30px;
}

table.docutils {
    border-collapse: collapse;
    border: 2px solid #aaa;
    margin: 0.5em 1.5em 0.5em 1.5em;
}

table.docutils td {
    padding: 2px;
    border: 1px solid #ddd;
}

p, li, dd, dt, blockquote {
    font-size: 15px;
    color: #333;
}

p {
    line-height: 150%;
    margin-bottom: 0;
    margin-top: 10px;
}

hr {
    border-top: 1px solid #ccc;
    border-bottom: 0;
    border-right: 0;
    border-left: 0;
    margin-bottom: 10px;
    margin-top: 20px;
}

dl {
    margin-left: 10px;
}

li, dt {
    margin-top: 5px;
}

dt {
    font-weight: bold;
}

th {
    text-align: left;
}

a {
    color: #990000;
}

a:hover {
    color: #c73f00;
}

pre {
    background-color: #f9f9f9;
    border-top: 1px solid #ccc;
    border-bottom: 1px solid #ccc;
    padding: 5px;
    font-size: 13px;
    font-family: Bitstream Vera Sans Mono,monospace;
}

tt {
    font-size: 13px;
    font-family: Bitstream Vera Sans Mono,monospace;
    color: black;
    padding: 1px 2px 1px 2px;
    background-color: #f0f0f0;
}

cite {
    /* abusing <cite>, it's generated by ReST for `x` */
    font-size: 13px;
    font-family: Bitstream Vera Sans Mono,monospace;
    font-weight: bold;
    font-style: normal;
}

#backlink {
    float: right;
    font-size: 11px;
    color: #888;
}

div.toc {
    margin: 0 0 10px 0;
}

div.toc h2 {
    font-size: 20px;
}
.syntax  { background: #ffffff; }
.syntax .c { color: #888888 } /* Comment */
.syntax .err { color: #a61717; background-color: #e3d2d2 } /* Error */
.syntax .k { color: #008800; font-weight: bold } /* Keyword */
.syntax .cm { color: #888888 } /* Comment.Multiline */
.syntax .cp { color: #cc0000; font-weight: bold } /* Comment.Preproc */
.syntax .c1 { color: #888888 } /* Comment.Single */
.syntax .cs { color: #cc0000; font-weight: bold; background-color: #fff0f0 } /* Comment.Special */
.syntax .gd { color: #000000; background-color: #ffdddd } /* Generic.Deleted */
.syntax .ge { font-style: italic } /* Generic.Emph */
.syntax .gr { color: #aa0000 } /* Generic.Error */
.syntax .gh { color: #303030 } /* Generic.Heading */
.syntax .gi { color: #000000; background-color: #ddffdd } /* Generic.Inserted */
.syntax .go { color: #888888 } /* Generic.Output */
.syntax .gp { color: #555555 } /* Generic.Prompt */
.syntax .gs { font-weight: bold } /* Generic.Strong */
.syntax .gu { color: #606060 } /* Generic.Subheading */
.syntax .gt { color: #aa0000 } /* Generic.Traceback */
.syntax .kc { color: #008800; font-weight: bold } /* Keyword.Constant */
.syntax .kd { color: #008800; font-weight: bold } /* Keyword.Declaration */
.syntax .kp { color: #008800 } /* Keyword.Pseudo */
.syntax .kr { color: #008800; font-weight: bold } /* Keyword.Reserved */
.syntax .kt { color: #888888; font-weight: bold } /* Keyword.Type */
.syntax .m { color: #0000DD; font-weight: bold } /* Literal.Number */
.syntax .s { color: #dd2200; background-color: #fff0f0 } /* Literal.String */
.syntax .na { color: #336699 } /* Name.Attribute */
.syntax .nb { color: #003388 } /* Name.Builtin */
.syntax .nc { color: #bb0066; font-weight: bold } /* Name.Class */
.syntax .no { color: #003366; font-weight: bold } /* Name.Constant */
.syntax .nd { color: #555555 } /* Name.Decorator */
.syntax .ne { color: #bb0066; font-weight: bold } /* Name.Exception */
.syntax .nf { color: #0066bb; font-weight: bold } /* Name.Function */
.syntax .nl { color: #336699; font-style: italic } /* Name.Label */
.syntax .nn { color: #bb0066; font-weight: bold } /* Name.Namespace */
.syntax .py { color: #336699; font-weight: bold } /* Name.Property */
.syntax .nt { color: #bb0066; font-weight: bold } /* Name.Tag */
.syntax .nv { color: #336699 } /* Name.Variable */
.syntax .ow { color: #008800 } /* Operator.Word */
.syntax .w { color: #bbbbbb } /* Text.Whitespace */
.syntax .mf { color: #0000DD; font-weight: bold } /* Literal.Number.Float */
.syntax .mh { color: #0000DD; font-weight: bold } /* Literal.Number.Hex */
.syntax .mi { color: #0000DD; font-weight: bold } /* Literal.Number.Integer */
.syntax .mo { color: #0000DD; font-weight: bold } /* Literal.Number.Oct */
.syntax .sb { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Backtick */
.syntax .sc { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Char */
.syntax .sd { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Doc */
.syntax .s2 { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Double */
.syntax .se { color: #0044dd; background-color: #fff0f0 } /* Literal.String.Escape */
.syntax .sh { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Heredoc */
.syntax .si { color: #3333bb; background-color: #fff0f0 } /* Literal.String.Interpol */
.syntax .sx { color: #22bb22; background-color: #f0fff0 } /* Literal.String.Other */
.syntax .sr { color: #008800; background-color: #fff0ff } /* Literal.String.Regex */
.syntax .s1 { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Single */
.syntax .ss { color: #aa6600; background-color: #fff0f0 } /* Literal.String.Symbol */
.syntax .bp { color: #003388 } /* Name.Builtin.Pseudo */
.syntax .vc { color: #336699 } /* Name.Variable.Class */
.syntax .vg { color: #dd7700 } /* Name.Variable.Global */
.syntax .vi { color: #3333bb } /* Name.Variable.Instance */
.syntax .il { color: #0000DD; font-weight: bold } /* Literal.Number.Integer.Long */
  


  
    Pygments
    Builtin Tokens
    
      « Back To Index
    
    
      
        Contents
        
        
          Keyword Tokens
        
          Name Tokens
        
          Literals
        
          Operators
        
          Punctuation
        
          Comments
        
          Generic Tokens
        
        
      
    
    
Inside the pygments.token module, there is a special object called Token
that is used to create token types.
You can create a new token type by accessing an attribute of Token:
>>> from pygments.token import Token
>>> Token.String
Token.String
>>> Token.String is Token.String
True

Note that tokens are singletons so you can use the is operator for comparing
token types.
As of Pygments 0.7 you can also use the in operator to perform set tests:
>>> from pygments.token import Comment
>>> Comment.Single in Comment
True
>>> Comment in Comment.Multi
False

This can be useful in filters and if you write lexers on your own without
using the base lexers.
You can also split a token type into a hierarchy, and get the parent of it:
>>> String.split()
[Token, Token.Literal, Token.Literal.String]
>>> String.parent
Token.Literal

In principle, you can create an unlimited number of token types but nobody can
guarantee that a style would define style rules for a token type. Because of
that, Pygments proposes some global token types defined in the
pygments.token.STANDARD_TYPES dict.
For some tokens aliases are already defined:
>>> from pygments.token import String
>>> String
Token.Literal.String

Inside the pygments.token module the following aliases are defined:







Text
Token.Text
for any type of text data

Whitespace
Token.Text.Whitespace
for specially highlighted whitespace

Error
Token.Error
represents lexer errors

Other
Token.Other
special token for data not
matched by a parser (e.g. HTML
markup in PHP code)

Keyword
Token.Keyword
any kind of keywords

Name
Token.Name
variable/function names

Literal
Token.Literal
Any literals

String
Token.Literal.String
string literals

Number
Token.Literal.Number
number literals

Operator
Token.Operator
operators (+, not...)

Punctuation
Token.Punctuation
punctuation ([, (...)

Comment
Token.Comment
any kind of comments

Generic
Token.Generic
generic tokens (have a look at
the explanation below)



The Whitespace token type is new in Pygments 0.8. It is used only by the
VisibleWhitespaceFilter currently.
Normally you just create token types using the already defined aliases. For each
of those token aliases, a number of subtypes exists (excluding the special tokens
Token.Text, Token.Error and Token.Other)
The is_token_subtype() function in the pygments.token module can be used to
test if a token type is a subtype of another (such as Name.Tag and Name).
(This is the same as Name.Tag in Name. The overloaded in operator was newly
introduced in Pygments 0.7, the function still exists for backwards
compatiblity.)
With Pygments 0.7, it's also possible to convert strings to token types (for example
if you want to supply a token from the command line):
>>> from pygments.token import String, string_to_tokentype
>>> string_to_tokentype("String")
Token.Literal.String
>>> string_to_tokentype("Token.Literal.String")
Token.Literal.String
>>> string_to_tokentype(String)
Token.Literal.String


Keyword Tokens

Keyword
For any kind of keyword (especially if it doesn't match any of the
subtypes of course).
Keyword.Constant
For keywords that are constants (e.g. None in future Python versions).
Keyword.Declaration
For keywords used for variable declaration (e.g. var in some programming
languages like JavaScript).
Keyword.Pseudo
For keywords that aren't really keywords (e.g. None in old Python
versions).
Keyword.Reserved
For reserved keywords.
Keyword.Type
For builtin types that can't be used as identifiers (e.g. int,
char etc. in C).



Name Tokens

Name
For any name (variable names, function names, classes).
Name.Attribute
For all attributes (e.g. in HTML tags).
Name.Builtin
Builtin names; names that are available in the global namespace.
Name.Builtin.Pseudo
Builtin names that are implicit (e.g. self in Ruby, this in Java).
Name.Class
Class names. Because no lexer can know if a name is a class or a function
or something else this token is meant for class declarations.
Name.Constant
Token type for constants. In some languages you can recognise a token by the
way it's defined (the value after a const keyword for example). In
other languages constants are uppercase by definition (Ruby).
Name.Decorator
Token type for decorators. Decorators are synatic elements in the Python
language. Similar syntax elements exist in C# and Java.
Name.Entity
Token type for special entities. (e.g. &nbsp; in HTML).
Name.Exception
Token type for exception names (e.g. RuntimeError in Python). Some languages
define exceptions in the function signature (Java). You can highlight
the name of that exception using this token then.
Name.Function
Token type for function names.
Name.Label
Token type for label names (e.g. in languages that support goto).
Name.Namespace
Token type for namespaces. (e.g. import paths in Java/Python), names following
the module/namespace keyword in other languages.
Name.Other
Other names. Normally unused.
Name.Tag
Tag names (in HTML/XML markup or configuration files).
Name.Variable
Token type for variables. Some languages have prefixes for variable names
(PHP, Ruby, Perl). You can highlight them using this token.
Name.Variable.Class
same as Name.Variable but for class variables (also static variables).
Name.Variable.Global
same as Name.Variable but for global variables (used in Ruby, for
example).
Name.Variable.Instance
same as Name.Variable but for instance variables.



Literals

Literal
For any literal (if not further defined).
Literal.Date
for date literals (e.g. 42d in Boo).
String
For any string literal.
String.Backtick
Token type for strings enclosed in backticks.
String.Char
Token type for single characters (e.g. Java, C).
String.Doc
Token type for documentation strings (for example Python).
String.Double
Double quoted strings.
String.Escape
Token type for escape sequences in strings.
String.Heredoc
Token type for "heredoc" strings (e.g. in Ruby or Perl).
String.Interpol
Token type for interpolated parts in strings (e.g. #{foo} in Ruby).
String.Other
Token type for any other strings (for example %q{foo} string constructs
in Ruby).
String.Regex
Token type for regular expression literals (e.g. /foo/ in JavaScript).
String.Single
Token type for single quoted strings.
String.Symbol
Token type for symbols (e.g. :foo in LISP or Ruby).
Number
Token type for any number literal.
Number.Float
Token type for float literals (e.g. 42.0).
Number.Hex
Token type for hexadecimal number literals (e.g. 0xdeadbeef).
Number.Integer
Token type for integer literals (e.g. 42).
Number.Integer.Long
Token type for long integer literals (e.g. 42L in Python).
Number.Oct
Token type for octal literals.



Operators

Operator
For any punctuation operator (e.g. +, -).
Operator.Word
For any operator that is a word (e.g. not).



Punctuation
New in Pygments 0.7.

Punctuation
For any punctuation which is not an operator (e.g. [, (...)



Comments

Comment
Token type for any comment.
Comment.Multiline
Token type for multiline comments.
Comment.Preproc
Token type for preprocessor comments (also <?php/<% constructs).
Comment.Single
Token type for comments that end at the end of a line (e.g. # foo).
Comment.Special
Special data in comments. For example code tags, author and license
informations etc.



Generic Tokens
Generic tokens are for special lexers like the DiffLexer that doesn't really
highlight a programming language but a patch file.

Generic
A generic, unstyled token. Normally you don't use this token type.
Generic.Deleted
Marks the token value as deleted.
Generic.Emph
Marks the token value as emphasized.
Generic.Error
Marks the token value as an error message.
Generic.Heading
Marks the token value as headline.
Generic.Inserted
Marks the token value as inserted.
Generic.Output
Marks the token value as program output (e.g. for python cli lexer).
Generic.Prompt
Marks the token value as command prompt (e.g. bash lexer).
Generic.Strong
Marks the token value as bold (e.g. for rst lexer).
Generic.Subheading
Marks the token value as subheadline.
Generic.Traceback
Marks the token value as a part of an error traceback.



  



